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GUIDELINES 

FOR 

RESEARCH  PLANNING  AND  DESIGN 

I.  TASK  ANALYSIS  AS  RESEARCH 

The  Task  Analysis  program  conducted  by  the  Office  of  Man- 
power utilization,  HQ,  USMC  (OMU) , was  established  to  discover, 
develop,  and  report  answers  to  questions  that  have  significance 
in  solving  Marine  Corps  management  problems.  To  this  end,  the 
Task  Analysis  program  searches  out  and  gathers  relevant  infor- 
mation, and  analyzes  these  data  to  provide  meaningful  answers 
to  specified  questions  about  what  Marines  actually  do  in  their 
various  assignments.  Thus  it  is  clear  that  the  mission  of  OMU 
is  to  conduct  research,  i.e.,  purposive,  systematic  investiga- 
tion and  analysis,  and  to  generate  reports  of  findings  that  will 
be  useful  and  influential  in  Marine  Co3:ps  planning,  policy  de- 
termination, and  management.  In  all  such  research,  the  credi- 
bility emd  influence  of  reports  and  recommendations  depend 
heavily  on  the  evident  logic,  carefulness  and  thoroughness  of 
research  planning  eind  design.  It  follows  that  if  OMU's  re- 
ports and  contributions  are  to  achieve  maximum  impact  and 
value,  the  planning  and  design  of  projects  roust  justify  res- 
pect and  high  credibility.  To  accomplish  this,  each  project 
must  have  been  planned,  designed  and  conducted  in  a manner 
consistent  with  widely  recognized  criteria  of  dependable 
scientific  research.  Such  criteria  have  beccxne  generally 
recognized,  accepted,  and  understood  in  eveiry  major  scienti- 
fic area. 

Research  in  Occupational  Task  Analysis  falls  in  the  dcxnain 
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of  the  behavioral  sciences.  This  is  so  because  it  studies 
human  behavior,  and  shares  not  only  methods  and  techniques  with 
B^avioral  Science,  but  also  because  it  in^licitly  and  express- 
ly strives  for  similar  goals.  While  Task  Analysis  does  not 
emphasize  the  goals  of  explemation  and  prediction,  it  parallels 
the  logical  processes  of  attaining  such  goals  up  through  the 
inteiq>retation  of  the  meaning  of  data.  Task  Analysis  strives 
to  make  valid  empirical  generalizations  about  the  behavior  of 
individuals  who  are  engaging  in  activities  in  their  natural  en- 
vironments . 

As  with  any  scientific  endeavor,  Task  Analysis  delimits 
the  types  of  activities  to  be  investigated  and  the  context 
(environment)  in  which  such  activities  are  to  be  meaningfully 
observed.  In  the  case  of  OMU,  these  activities  are  defined 
as  tasks  that  relate  to  the  performance  of  duties  in  the  con- 
text in  which  they  are  legitimately  performed.  The  purpose 
of  this  report  is  to  suggest  a framework  for  Task  Analysis 
which  formally  recognizes  its  parallels  and  similarities  with 
the  methodology  of  the  Behavioral  Sciences  and  contributes  to 
the  production  of  high  quality  analysis  and  reporting.  By 
making  such  a framework  explicit,  it  is  possible  to  standardize 
many  of  the  steps  in  the  process  of  OMU  research.  It  addition- 
ally provides  a system  for  internal  evaluation  of  research 
findings  and  results. 

Specifying  phases  and  rationale.  Well-designed  research 
begins  with  a preliminary  outline  and  definition  of  the  specific 


steps  or  phases  to  be  undertaken,  together  with  an  explicit 
statement  of  the  accompanying  rationale  for  each  step. 

Analysis  is  not  an  activity  unique  to  scientists.  Every 
individual  analyzes  things  many  times  a day.  The  difference 
between  a scientist-analyst  and  a non-professional  analyst  is 
that  the  scientist  specifically  and  consciously  states  the 
steps  and  rationale  before  and  throughout  his  analysis.  That 
is,  there  is  an  explicit  method  to  his  madness.  It  does  not 
mean  that  his  ultimate  conclusions  are  necessarily  any  more 
correct  than  those  of  the  layman  (although  they  usually  should 
be) . What  it  does  mean  is  that  any  other  adequately  trained 
professional  can  perform  the  same  process  and  reach  the  same 
conclusion.  If  the  same  conclusion  cannot  be  reached,  then 
another  scientist  can  check  the  logical  steps  and  instr\iments 
used  which  led  to  the  first  conclusion  and  challenge  them  on 
the  basis  of  scientific  merit,  rather  than  on  personal  and 
subjective  feeling. 

Part  of  the  problem  in  non-scientific  analysis  is  that 
many  of  the  phases  of  the  research  are  not  made  explicit. 

This  has  been  an  issue  in  the  social  and  behavioral  sciences 
for  years.  Only  in  the  last  generation  has  there  been  much 
systematic  effort  made  to  counter  this  trend.  In  Anthropology 
and  Sociology  particularly,  the  problem  has  become  well-recog- 
nized. Beginning  with  some  broad,  general  topic,  researchers 
have  often  gone  into  previously  unknown  societies  (or  sub- 
groups in  their  own  society)  and  have  tried  to  absorb  and 
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understand  everything  they  see  and  hear.  This  naive  and  un- 
structured approach  has  sometimes  been  necessary  in  order  to 
form  the  groundwork  for  a more  problem-oriented  analysis. 
Essentially,  these  analysts  have  been  trying  to  learn  to  "think 
like  an  Indian"  without  knowing  what  such  an  activity  really 
means.  However,  the  recording  of  such  observations  has  been 
of  value.  Basic  data  have  now  been  collected  so  that  con- 
temporary researchers  can  gain  much  of  the  background  infor- 
mation necessary  for  their  studies  through  books,  formal 
training,  and  informal  conversations  with  individuals  experi- 
enced in  their  fields  of  interest.  Such  reliance  on  existing 
resources  not  only  saves  valuable  research  time,  but  it  also 
frequently  results  in  superior  research  designs,  data  collec- 
tion instruments,  techniques,  and  analyses. 

OMU  can  be  said  to  be  in  a similar  position.  Beginning 
with  a slight  understanding  of  the  problem,  lack  of  experience 
in  how  to  study  the  problem,  and  uncertainty  as  to  how  to  in- 
terpret and  analyze  the  results,  OMU  now  has  reached  a point 
where  it  can  benefit  from  the  experience  of  past  analysts  and 
analyses.  While  there  are  undoubtedly  some  errors  to  be  dis- 
covered in  past  studies,  these  studies  none-the-less  provide 
a background  of  analytic  techniques  and  methodological  orienta- 
tions on  which  future  research  can  be  based.  While  it  is  not 
necessarily  true  that  "experience  is  the  best  teacher,"  it  is 
the  unwise  researcher  who  ignores  it. 

This  report  proposes  to  provide  OMU  with  an  Analytical 
Philosophy,  or  RESEARCH  DESIGN  that  is  general  enough  to  be 
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used  for  any  particular  study.  PART  II  discusses  the  qualifi- 
cations of  the  researcher.  PART  III,  the  bulk  of  the  report, 
describes  the  elements  in  research  planning  and  design,  noting 
especially  the  requirements  for  a definitive  statement  of  the 
problem,  the  potential  contributions  of  theory,  and  the  process 
of  operationalizing,  including  the  literature  search,  observa- 
tions and  interviews,  measuring  instruments,  and  guidelines 
for  sampling,  data  collection,  analysis,  and  reporting.  In 
overview,  this  report  discusses  a variety  of  technical  questions 
such  as  PROBLEM  DEFINITION,  CONCEPTS  AND  OPERATIONAL  DEFINITIONS, 
OPERATIONS,  HYPOTHESES,  AND  EVIDENCE.  It  proposes  a format  for 
the  systematic  study  of  an  Occupational  Field.  Additionally, 
and  in  some  respects  more  importantly,  it  discusses  the  role  of 
the  researcher  in  the  process  of  research. 

II . THE  RESEARCHER 


The  key  factor  in  research  is  the  researcher  himself.  No 
technique,  method  or  reference  material  is  so  crucial  and  cri- 
tical to  the  quality  of  the  research  as  are  the  efforts  and 
objectivity  of  the  individual  researcher.  In  general  terms, 
what  is  a researcher?  He  is  a product  of  a lifetime  of  experi- 
ences. attitudes,  beliefs,  biases,  and  opinions.  He  has  tech- 
nical knowledge  in  some  field  of  activity,  whether  this  be  com- 
puter science,  auto  mechanics,  physics,  biology  or  carpentry  or 
some  other . He  has  spent  his  lifetime  learning  about  the  world 


around  him  and  has  formed  opinions  and  theories  regarding  how 
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it  functions  and  why.  He  has  his  own  private  philosophy,  per- 
haps his  own  religion,  and  his  own  interests.  Because  of  this, 
it  is  unrealistic  to  pretend  that  when  he  puts  on  his  research 
hat  he  automatically  becomes  a completely  objective  observer 
of  some  phenomenon.  His  life  experiences  remain  very  real,  and 
they  guide  his  research  efforts.  And  this  is  how  it  should  be. 
In  itself,  this  is  neither  good  nor  bad,  but  life  experiences 
can  be  used  poorly  or  well. 

In  order  to  use  his  personal  qualities  to  his  best  advan- 
tages, the  researcher  must  make  explicit  some  of  his  personal 
feelings  and  beliefs  so  that  he  is  consciously  aware  of  what 
they  are  and  how  they  may  affect  the  way  he  goes  about  his  task. 
This  is  especially  critical  in  behavioral  or  social  research 
where  our  feelings  about  the  nature  of  man  and  our  view  of  the 
world  tend  to  be  relatively  unscientific  and  personal.  It 
makes  little  difference  in  automotive  design,  for  excurple,  if 
the  engineer  is  a Republican  or  a Democrat.  However,  if  a 
political  scientist  is  engaged  in  predicting  election  results, 
such  a bias  should  be  recognized  in  his  research  plan  so  that 
he  would  not  make  mistakes  in  constructing  his  questionnaire, 
selecting  his  sample,  hiring  and  training  interviewers  and 
drawing  conclusions  frcxn  his  data. 

Further,  a researcher  has  a certain  amount  of  formal  and 
infoxmal  knowledge  about  various  things.  At  OMU,  for  example, 
several  different  technical  specialties  are  represented  among 
members  of  its  staff.  OMU  analysts  share  a certain  degree  of 
knowledge  with  the  people  they  study.  This  can  be  general 
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military  knowledge,  technical  knowledge,  or  a combination  of 
both.  VJhen  one  Marine  meets  another,  and  they  discuss  their 
jobs,  a certain  amount  of  knowledge  is  informally  transmitted. 

As  a consequence,  it  is  rare  to  find  a Marine  who  has  no  knowl- 
edge v^atever  of  what  someone  does  in  Motor  Transport,  Tanks, 
or  Intelligence.  The  point  to  be  made  here  is  that  it  is  naive 
to  assume  that  there  is  such  a thing  as  a completely  naive  ap- 
proach to  Task  Analysis.  Rather  than  trying  to  mimic  the  ostrich 
and  hide  one's  head  in  the  sand,  OMU  needs  to  make  critical  use 
of  the  broad  range  of  knowledge  its  analysts  may  have  with  res- 
pect to  a particular  Military  Occupational  Specialty  (MOS)  or 
Occupational  Field  (OF) . Such  knowledge  can  provide  valuable 
insights  both  in  instrximent  construction  and  analysis.  The 
issue  here  refers  back  to  what  was  mentioned  regarding  the  ex- 
change of  knowledge  by  researchers  in  the  behavioral  sciences. 

If  it  were  necessary  to  gain  information  about  a society  or  an 
institution  only  by  means  of  primary  sources,  research  would 
not  be  additive,  and  there  would  be  little  productive  problem- 
oriented  research  conducted  by  behavioral  scientists  today. 

In  summary,  the  researcher  should  be  aware  of  personal 
attitudes  and  feelings  that  can  color  his  perceptions  and  inter- 
pretations of  data  developed  in  each  particular  project  or  study. 
He  should  further  understand  that  neither  he  nor  any  of  his  co- 
workers is  completely  naive.  OMU  administrators  can  benefit 
from  similar  understanding  and  can  use  the  background  of  knowl- 
edge and  insights  represented  in  its  cadre  of  researchers. 


8 


III.  RESEARCH  PLANNING  AND  DESIGN 

Broadly,  a Research  Design  is  a set  of  interrelated  plans 
which  researchers  develop  and  use  in  order  to  solve  a problem. 

It  is  much  the  same  as  the  plans  for  constructing  a building 
or  an  airplane,  or  for  getting  to  the  moon.  A Research  Design 
specifies  the  aims  and  objectives  of  the  research,  the  materials 
and  the  tools  and  information  needed,  the  sequence  or  order  in 
which  the  materials  are  assembled,  and  the  standards  against 
which  the  results  are  to  be  judged. 

The  elements  of  a good  research  design  are  broad  enough  to 
be  applicable  to  almost  any  field  of  scientific  inq[uiry.  In 
outline  form,  they  follow  the  general  and  sequential  pattern 
below: 

A.  Statement  of  the  Research  Problem — Hypotheses.  Before 
any  true  research  activity  can  be  undertaken,  the  problem  which 
generates  the  research  must  be  stated  explicitly.  It  is  not 
enough  to  say  that  we  want  to  know  about  the  manpower  require- 
ments of  an  occupational  field.  However,  that  is  a good  start- 
ing point  for  the  specific  formulation  of  a problem.  We  need 
to  specify  beyond  that  exactly  WHAT  we  want  to  know  adjout  man- 
power in  an  Occupational  Field  (a  problem)  and  WHY  we  want  to 
know  it  (a  rationale) . 

Often  the  central  HYPOTHESIS  is  buried  in  the  statement  of 
the  problan.  For  example,  it  may  be  that  a fair  general  state- 
ment of  the  problem  at  OMU  is: 
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To  what  degree  do  the  current  structure  of 
of  OFxx  and  the  training  of  Marines  with  a desig- 
nated MOS  in  that  field  actually  meet  the  real 
needs  required  to  fulfill  the  operational  mission 
of  that  OP?  And:  if  they  do  not  do  so  adequately, 

how  can  deficiencies  be  overcome? 

We  can  state  the  hypothesis  formally  as  follows: 

OFxx  is  staffed  and  Marines  are  trained  in 
such  a way  that  the  current  personnel  structure 
and  training  are  adequate  to  accomplish  the  mis- 
sion of  the  OF. 

Alternatively : 

The  staffing  and  training  of  personnel  in 
OFxx  are  not  adequate  to  accomplish  its  mission. 

When  the  problem  is  stated  in  such  a fashion,  it  is  far 
easier  to  plan  and  operationalize  research  in  order  to  deter- 
mine which  of  the  two  (or  more)  competing  hypotheses  is  correct. 

B.  Contribution  of  Theory.  A theory  is  a set  of  plausible 
propositions  that  appear  to  explain  and  predict  certain  phenom- 
ena. In  the  Marine  Corps  there  exists  a set  of  propositions 
\idiich  tends  to  do  this  as  well.  While  not  formally  stated  or 
recognized  as  a theory,  it  in  fact  governs  some  behaviors  of 
some  individuals  under  some  circumstances.  It  is  not  an  ele- 
gant theory,  and  it  falls  short  of  many  demands  placed  upon 
other  theories,  but  none-the-less,  it  exists  and  functions  to 
some  degree.  We  have  chosen  to  call  this  the  "CAN  DO"  THEORY. 

It  has  its  roots  in  the  indoctrination  during  recruit  and 
officer  training;  it  is  formally  recognized  as  being  of  social 
value  to  the  Marine  Corps;  and,  it  is  acted  upon  so  frequent- 
ly that  we  can  expect  it  to  be  of  value  in  solving  part  of  the 
research  problem  stated  above. 
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That  is  one  of  the  criteria  of  a theory.  It  exists  to  help 
solve  problems.  If  it  were  to  be  stated  more  formally,  the  CAN 
DO  theory  would  probably  go  something  like  this: 

1.  The  Marine  Corps  has  a mission  to  fulfill. 

2.  The  Marine  Corps  has  traditionally  fulfilled 
this  mission  under  both  optimal  and  sub-opti- 
mal circumstances. 

3.  OFxx  is  part  of  the  Marine  Corps  and  has  its 
own  support  mission  to  fulfill. 

4.  OFxx  will  probably  fulfill  its  mission. 

From  these  statements  one  can  begin  to  operationalize 
research  on  the  problem  based  on  the  CAN  DO  theory.  Stating 
this  in  the  form  of  a hypothesis  we  have: 

If  OFxx  is  to  fulfill  its  mission,  then  it 

must  have  a personnel  structure  which  permits  it 

to  do  so. 

If  it  has  the  personnel  structure  to  do  so, 

then  that  structure  exists  because: 

a.  It  is  properly  staffed  and  trained,  or 

b.  It  is  improperly  staffed  and  trained, 
but  deficiencies  in  training  and  staf- 
fing are  made  up  for  informally,  and 
the  structure  still  exists  and  the 
mission  is  accomplished. 

As  indicated  above,  this  is  not  the  most  elegant  theory 
science  has  seen,  but  it  probably  is  not  the  worst  either. 

The  fact  is  that  often  the  proof  of  the  pudding  is  in  the 
eating,  and  if  the  theory  serves  to  lighten  the  difficulties 
of  the  research  process  and  bring  about  correct  and  efficient 
findings,  then  it  stands  on  its  own  merit. 

C.  Operationalizing  the  Research.  Once  the  problem  has 
been  specified,  and  a theoretical  guidance  or  direction  (Hypoth- 
esis) has  been  established,  it  is  necessary  to  begin  searching 
for  ways  in  which  the  hypothesis  can  be  tested.  There  are 
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several  ways  to  do  this  and  they  are  often  used  jointly. 

1.  Review  of  the  Literature.  In  OMU's  case,  this 
involves  searching  through  Tables  of  Organization  and  training 
Manuals,  etc.,  in  order  to  find  exactly  what  the  nature  and 
mission  of  the  Occupational  Field  is.  It  is  at  this  point 
that  someone  familiar  with  the  field  can  provide  valuable 
clues  and  leads  to  the  OMU  investigation. 

2.  Observation  and  Interview.  In  order  to  supple- 
ment the  review  of  literature,  it  is  necessary  to  find  out 
what  it  is  that  the  incumbents  in  this  field  actually  do. 

There  are  several  ways  for  carrying  out  this  assignment  and 
many  of  them  are  discussed  in  reports  prepared  in  Research 
Area  1 Observation  and  Interview  and  Research  Area  2 Task  In- 
ventory Construction.^  In  addition,  a source  that  should  not 
be  overlooked,  and  which  should  be  quite  reliable,  is  made 

up  of  local  records.  Such  records  as  Work  Order  forms,  per- 
sonnel turn-over  jackets  and  other  local  forms,  e.g.,  man- 
hour reports,  can  all  be  used  to  improve  the  operationaliza- 
tion of  the  research. 

3.  Measurement  Instruments.  Once  the  basic  back- 
ground is  obtained,  the  instruments  for  measurement  can  be 
designed.  Again,  other  Research  Areas  in  the  Project  have 
striven  to  provide  guidelines  on  this,  and  it  is  referenced 
here  to  maintain  the  logical  sequence  of  the  research  desion. 

Together,  these  steps  can  be  called  Operationalization . 
This  is  the  process  by  which  the  broad  concepts  delineated  in 
the  Statement  of  the  Problem  and  Theoretical  Contribution 


1.  See  Technical  Reports  No.  2 and  No.  11. 
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(Hypothesis)  are  reduced  to  measurable  variables. 

No  hypothesis  can  be  tested  as  valid  or  invalid  until  the 
concepts  it  articulates  are  operationalized  and  measured  in  the 
form  of  variables.  Very  often  this  is  the  real  challenge  in 
designing  research.  How,  in  fact,  does  one  reduce  a concept 
such  as  "current  personnel  structure"  to  a variable  or  group 
of  measurable  variables?  What  is  "adequate".  What,  in  fact, 
is  a "task",  and  how  does  it  differ  from  an  "element"  or  a 
"job"?  These  are  salient  issues  in  the  Operationalization  stage 
of  the  research,  and  here  is  where  most  problems  arise.  It  is 
at  this  point  that  an  individual's  personal  background  is  like- 
ly to  help  or  hinder.  It  is  precisely  at  this  stage  in  the  pro- 
cess of  research  where  it  is  most  necessary  to  explicate  the 
researcher's  rationales  for  decisions.  It  is  at  this  point 
that  the  greatest  objectivity  and  creativity  are  necessary.  And 
it  is  just  at  this  point  that  there  are  no  "cookbook"  guides 
available  for  researchers,  especially  in  Task  Analysis. 

In  many  other  research  endeavors  there  are  standardized 
inventories,  check-lists,  tests,  and  questionnaires  that  have 
been  checked  for  validity  and  reliability  and  are  available  to 
the  researcher.  However,  since  Task  Analysis  deals  with  a 
unique  problem  in  the  study  of  each  Occupational  Field,  such 
"pre-cooked",  standardized  instruments  are  not  available.  As 
a consequence  of  all  this,  it  is  necessary  to  be  most  careful 
and  most  critical  of  the  statements  that  are  selected  for  in- 
clusion in  the  inventory.  The  ideal  method  of  resolving  this 
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problem  would  be  to  conduct  a PRE-TEST  on  a small  proportion  of 
the  incumbents  in  an  OF.  Often,  however,  this  is  not  feasible, 
and  the  final  decision  about  what  to  include  or  exclude  must  be 
made  subjectively. 

At  this  point,  behavioral  scientists  studying  individuals 
in  their  natural  environments  rely  heavily  on  KEY  INFORMANTS, 
These  are  individuals  whom  the  researcher  has  reason  to  believe 
have  extensive  knowledge  of  the  subject  area  of  the  study.  Be- 
havioral scientists  go  to  key  informants  and  have  them  review 
their  questionnaires  and  inventories.  They  ask  them  to  delete 
items  which  are  unclear  or  redundant,  or  to  help  re-word  items 
to  make  them  more  understandable.  It  may  be  that  OMU  can  obtain 
the  expert  opinion  of  several  of  these  key  informants  who  are  in 
operational  posts  in  order  to  have  them  assess  the  inventory 
statements.  In  any  case,  in  terms  of  research  design  generally, 
this  is  one  of  the  most  critical  issues  of  concern  to  any  re- 
searcher. Do  the  questions  truly  measure  the  concept? 

D.  Determining  Sample  Size.  Sample  size  is  often  deter- 
mined in  conjunction  with  the  earlier  stages  of  the  design,  and 
is  placed  in  this  position  as  a separate  step  because  of  its 
complexity.  Research  Area  3 has  prepared  guidelines  for  this 
portion  of  the  research  design,  (See  Technical  Report  No.  12) 
and  it  would  be  redundant  to  repeat  them  here.  The  crucial 
point  to  recognize  is  that  often  there  may  be  non-statistical 
reasons  for  modifying  a decision  about  sample  size.  This  may 
be  due  to  monetary  or  time  concerns,  availability  of  subjects, 
or  many  other  factors.  However,  when  a decision  regarding  the 
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size  and  type  of  a sample  is  made,  the  rationale  for  that  deci- 
sion should  be  spelled-out  clearly,  along  with  the  strategy  by 
which  the  sample  is  to  be  obtained.  One  of  the  most  damaging 
attacks  that  can  be  made  against  otherwise  well-conceived  and 
operationalized  research  is  to  point  out  flaws  in  its  sampling 
design  or  techniques.  A large  sample  is  not  necessarily  a 
good  sample,  and  a small  sample  is  not  necessarily  a bad  one. 
What  makes  a sample  good  or  bad  is  how  well  it  meets  the  needs 
for  answering  the  research  questions.  (See  Technical  Report 
No.  12.) 

E.  Collecting  the  Data.  This  is  a relatively  straight- 
forward portion  of  the  research  process.  However,  an  important 
requirement  is  that  the  individual  who  administers  the  inven- 
tory should  avoid  any  actions  which  might  create  bias  in  the 
answers  of  respondents.  Often  such  “minor “ factors  as  tone  of 
voice,  facial  expression,  posture,  yawning,  impatience,  etc., 
can  completely  ruin  the  collection  of  data.  Even  the  choice 
of  setting  for  administration  can  affect  the  results.  Another 
concern  which  parallels  this  is  unifonnitv  of  instructions.  If, 
for  example,  questions  are  likely  to  arise  about  what  is  an 
“average  amount  of  time, “ such  definitions  should  be  made 
clear  to  all  administrators  before  any  inventories  are  admin- 
istered. Again,  this  means  spelling-out  exactly  what  the  re- 
searcher means  by  such  a phrase.  And  again,  it  involves  much 
of  the  researcher's  background  and  personality.  It  is  clearly 
the  case  that  “average"  often  means  different  things  to  dif- 


ferent individuals.  In  order  to  assure  that  answers  are 
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comparable,  a common  definition  or  standard  must  be  specified. 
preferably  in  writing,  and  read  aloud  to  respondents. 

F.  Analyzing  the  Data.  A research  program  must  always  be 
designed  with  the  type  of  data-analysis  techniques  to  be  used 
clearly  in  mind.  As  a result,  the  researcher  must  constantly 
make  sure  that  his  findings  meet  the  reguirements  imposed  by 
his  analytical  techniques.  This  is  especially  important  when 
certain  types  of  statisrical  analysis  are  to  be  used.  Data 
must  be  appropriate  for  these  analytical  procedures . In  the 
case  of  OMU,  there  is  a pre-existing  analytical  package  of  com- 
puter programs  specifically  designed  for  Task  Analysis.  This 
Comprehensive  Occupational  Data  Analysis  Program  (CODAP)  con- 
tains numerous  sub-routines  which  provide  summary  information 
about  task  inventory  data.  It  is  essential  that  data  analysts 
at  OMU  be  completely  familiar  with  the  power  and  limitations  of 
such  routines  so  that  they  are  neither  under-used,  nor  stretched 
beyond  their  logical  limits. 

As  has  been  stressed  throughout  these  guidelines,  it  is  of 
utmost  importance  to  keep  clearly  in  mind  the  purpose  of  the  re- 
search (i.e.,  the  statement  of  the  problem).  At  each  step  in 
the  analysis  each  team  should  ask,  "Is  this  helping  to  answer 
the  questions  posed  by  the  research  problem?" 

There  are  probably  as  many  valid  approaches  to  data  an- 
alysis as  there  are  researchers,  even  given  the  relative  rigidity 
of  the  types  of  techniques  used.  As  with  the  operationalization 
phase,  the  data  analysis  phase  is  a reflection  of  the  character- 
istics of  the  researchers.  All  of  their  knowledge  comes  into 
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play  at  this  point.  Hunches  and  educated  guesses  often  suggest 
clues  to  relationships  v^ich  otherwise  would  have  gone  unno- 
ticed. 

However,  along  with  such  freedom,  there  are  restrictions 
which  must  be  observed.  For  each  and  every  inference  beyond  the 
raw  data,  a rationale  must  be  stated.  It  is  clear  that  the  fur- 
ther an  analyst  departs  from  the  basic  data  the  more  tenuous  the 
links  of  logic  become.  Because  of  this,  it  is  necessary  to  de- 
fine clearly  each  step  to  be  taken  by  setting  forth  the  assump- 
tions implicit  in  the  inferential  process,  the  rationale  that 
supports  the  inference,  and  the  logical  limitations  which  con- 
strain the  inference. 

At  the  heart  of  data  analysis  in  OMU  is  the  heirarchical 
clustering  program.  The  basic  purpose  of  this  routine  is  to 
find  Marines  who  share  certain  similarities  in  tasks  performed 
and  group  them  together  so  that  they  form  distinctive  categories 
that  are  different  from  all  other  clusters  and  individuals  in 
the  sample. 

Clustering  or  classification  analysis  is  an  activity  per- 
formed by  every  individual  throughout  the  course  of  his  life. 

It  is  a necessary  function  of  human  activity  that  things  be 
classified  in  some  systematic  way  so  as  to  include  those  that 
are  alike  and  exclude  other  things.  Take  the  simple  example  of 
food:  What  is  food?  What  is  non-food?  If  any  human  wants  to 

survive,  he  must  be  able  to  distinguish  foods  from  non-foods. 
Scientists  in  every  field  also  are  vitally  concerned  with  making 
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similar  distinctions.  As  a result  of  the  complexity  and  varia- 
bility of  the  objects  which  surround  us,  it  has  been  necessary 
to  find  some  mathematical  wav  of  clustering  things  so  that  the 
process  is  logical  and  objective.  Subjective  classifications 
tend  to  lack  consistency  and  uniformity,  and  thus  introduce 
errors  because,  as  pointed  out  above,  the  researcher  has  nvunerous 
biases  of  which  he  is  frequently  not  even  aware.  Recognition  of 
this  tendency  has  led  to  considerable  study  of  the  possible 
rationales  and  devices  for  objective  clustering.  (References  on 
clustering  and  classification  analysis  are  listed  in  the  selected 
bibliography  at  the  end  of  this  report. 

OMU  has  adopted  some  rule-of-thumb  guidelines  for  deter- 
mining objectively  the  critical  inclusion  level  for  stages  or 
composites.  The  35%,  50%  homogeneity  levels  within  and  between 
stages  is  said  to  have  been  empirically  derived  as  being  the 
most  efficient.  However,  it  is  often  the  case  that  such  mea- 
sures are  not  meaningful  since  the  "true"  cluster  for  a job 
type  includes  stages  which  fall  short  of  these  optimxim  figures. 
There  is  a sound  mathematical  reason  for  this.  Principally, 
it  is  the  result  of  a very  large  number  of  items  in  an  inven- 
tory . As  the  number  of  items  increases,  the  opportunity  for  an 
incumbent  to  respond  to  a large  number  of  items  is  also  increased. 
The  greater  the  nximber  of  responses,  the  smaller  the  "time  spent" 
and  "shared  time"  spent  percentages  for  any  item  become.  Conse- 
quently, when  the  matrix  is  searched  for  the  time  similarity 
between  and  eunong  incumbents,  the  probability  is  reduced  that 


1.  Also  see  Technical  Report  No.  1,  HIERARCHICAL  CLUSTERING: 
A BIBLIOGRAPHY. 
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any  two  or  more  individuals  will  manifest  a high  percentage  of 
similarity.  As  a result,  the  principle  of  35%  and  50%  may  not 
hold,  and  lesser  levels  may  be  required  for  decisions  on  in- 
clusion or  exclusion. 

Recognizing  this,  the  researcher  must  excunine  all  reason- 
able alternatives,  and  he  should  explicitly  state  why  each  and 
every  decision  was  made  at  a particular  stage.  The  consequences 
for  ignoring  the  need  for  such  a justification  can  be  severe. 

For  example,  the  results  of  an  otherwise  solid  research  design 
may  be  jeopardized.  Assume  that  a decision  was  made  to  include 
a particular  sub-group  in  some  cluster,  and  no  specific  justifi- 
cation was  stated.  At  later  stages  in  the  analysis,  that  group 
(vAiich  in  reality  belongs  somewhere  else)  is  not  available  for 
inclusion  in  its  appropriate  place  because  it  has  already  been 
classified.  This  in  turn  affects  the  decisions  made  about  other 
clusters  of  job  types,  resulting  in  a distorted  analysis  of  the 
entire  Occupational  Field.  Such  cximulative  errors  require 
extensive  trace-back  time,  and  it  is  often  impossible  to  identify 
successfully  the  location  of  the  original  error  if  justifications 
and  rationales  for  each  decision  level  are  not  specified.  Too 
mcinv  personal  variables  enter  into  subjective  decisions  to  be 
accurately  recalled  at  a later  time.  For  example,  how  often  do 
many  of  us  think  back  in  time  and  say,  "Now  why  would  I do  a thing 
like  that?  It  makes  no  sense  at  all."  If,  on  the  other  hand, 
justification  for  each  decision  is  recorded,  the  analyst  can 
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readily  trace  back  the  source  of  his  error  and  rectify  it. 

Another  problem  which  can  plague  cluster  analysis  is  the 
issue  raised  by  isolets.  i.e.,  individuals  who  do  not  seem  to 
fit  in  any  of  the  clusters.  There  are  two  basic  ways  to  view 
this  phenomenon.  One  is  the  manner  in  which  OMU  is  currently 
treating  it.  That  is,  isolets  are  considered  as  valid  cases  and 
are  permitted  to  fall  where  they  may  in  the  clustering  procedure. 
Often  these  fall  out  toward  the  "end"  of  the  tree  diagram,  but 
they  may  be  included  elsewhere  and  distort  the  sub-group  into 
which  they  are  joined.  On  the  other  hand,  they  can  be  treated 
as  deviates . In  this  case,  they  are  assumed  to  be  nonsense 
responses,  i.e.,  attempts  to  deliberately  mislead,  or  the 
result  of  misunderstanding  some  items,  or,  as  highly  special- 
ized cases.  Under  this  philosophy  it  is  wise  to  extract  the 

cases  before  clustering  and  decide  if  they  are  deviates  or 
2 

special  cases.  In  either  instance,  they  are  analyzed  separate- 
ly for  the  information  they  contain.  If  they  are  deemed  special 
cases,  then  the  appropriate  analytical  approach  is  to  include 
them  as  CASE  STUDY  material  in  the  research  report.  Case 
study  is  a powerful  tool  in  analysis  and  should  not  be  ignored. 

In  order  to  determine  if  there  are  such  types  of  cases  in 
the  data  before  clustering,  a simple  procedure  is  recommended. 
Cross-tabulation  provides  a visual  representation  of  responses 
by  some  criterion  category.  If  the  inventory,  for  example. 


2.  For  a more  thorough  discussion  of  this  phenomenon,  see 
Everitt,  Brian,  CLUSTER  ANALYSIS,  New  York;  Wiley,  1974,  on 
"Outliers" . 


20 


contains  a number  of  tasks  which  usually  fall  in  officer  jobs, 
then  it  is  a simple  matter  to  sum  the  responses  by  these  jobs 
and  cross-tabulate  them  by  rank.  If  one  finds  E-4’s  and  E-5*s 
answering  a great  number  of  tasks  in  the  officer  job  categories, 
these  cases  can  be  examined  for  their  worth  before  including 
them  in  the  clustering  routine.  One  OF  examined  showed  an  E-4 
performing  21  officer-type  tasks.  This  method  of  cross-tabulat- 
ing certain  variables  is  an  excellent  way  of  searching  the  data 

3 

for  unusual  response  patterns  of  any  sort. 

Another  problem  in  data  analysis  is  when  to  stop.  Often 
the  data  reveal  numerous  unforeseen  relationships,  and  the 
conscientious  researcher  naturally  desires  to  pursue  the  reasons 
for  these.  There  comes  a time,  however,  when  this  activity  must 
be  terminated  and  the  write-up  stage  begun.  This  again  requires 
the  judgement  of  the  researcher.  The  appropriate  questions  to 
ask  are:  "Have  I answered  the  research  question  as  completely 

as  possible?"  "Is  there  any  further  information  which  will  im- 
prove the  research  report?"  "How  much  will  the  further  analysis 
add?"  These  are  questions  which  can  only  be  dealt  with  by  the 
research  team  that  has  labored  throughout  the  process.  Cut- 
ting-off (or  extending)  an  analysis  requires  a clearly  stated 
rationale.  Obviously  no  set  of  data  is  ever  ccxapletely  analyzed. 


3.  CODAP  does  not  have  an  adequate  cross-tabulation  routine. 
However,  several  are  readily  available  from  other  standard 
sources.  For  example  the  IBM  Statistical  Sub-routine  Package 
(SSP)  can  be  readily  used  with  this  type  of  data. 
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There  is  always  a stone  left  unturned  or  a parameter  left  unin- 
vestigated. But  the  crucial  issue  lies  at  the  heart  of  the  re- 
search. the  answer  to  a properly  stated  research  problem.  It 
is  always  possible  to  re-analyze  data,  but  in  many  scientific 
activities  it  is  important  to  find  and  report  the  results  as 
quickly  as  possible.  In  the  case  of  OMU,  delay  can  result  in 
a lag  in  management  decision-making.  In  basic  and  applied 
science,  a delay  means  postponing  the  dissemination  of  valuable 
information  to  colleagues  who  may  be  able  to  build  their  own  re- 
search on  the  findings.  However,  time  expediency  is  not  a suf- 
ficient rationale  for  terminating  data  analysis.  We  must  return 
to  the  basic  issue:  Does  what  we  have  satisfy  the  recruiremcnts 

of  the  research  problem? 

G.  The  Research  Report.  The  product  of  all  the  blood 
and  sweat,  excitement  and  boredom  of  the  reseach  process  is 
the  research  report.  Of  what  does  it  consist  and  how  is  it 
organized? 

Essentially,  the  research  report  puts  flesh  on  the  research 
design.  Conventionally,  the  report  re-states  the  problem,  theory, 
operations,  sampling  and  collection  procedures,  and  the  analyti- 
cal process.  However,  it  also  specifies  all  the  decision  ration- 
ales which  led  to  the  conclusions  and  recommendations,  in  other 
words,  the  report  should  be  a faithful  record  of  the  researchers' 
logic  from  the  problem  formation  to  the  conclusions.  It  is  the 
basis  upon  which  the  results  and  the  researchers  are  judged. 

The  more  thorough  and  painstaking  the  report,  the  greater  its 
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credibility  and  impact  and  the  probability  that  its  recommend- 
ations will  be  acted  upon  by  recipient  policy-makers. 

This  is  the  central  issue  of  credibility.  It  is  not  that 
the  findings  be  consistent  with  Marine  Corps  Manpower  philoso- 
phy and  doctrine  that  is  crucial.  Rather,  it  is  the  DEPTH  and 
STRENGTH  of  the  analysis  and  the  rationale  of  findings  that  are 
critical.  If  each  decision  is  justified  by  data;  if  the  steps 
in  analysis  are  specified;  if  solid  rationale  is  presented  for 
sampling  strategies;  if  the  operations  clearly  represent  measures 
of  the  concepts;  and.  if  the  problem  is  well  defined  and  stated, 
then  the  researcher  can  be  reasonably  sure  that  his  product  will 
receive  acceptance  and  serious  consideration.  It  is  when  these 
criteria  are  unmet  or  are  violated  that  policy-makers  justifi- 
ably ignore  or  criticize  the  product.  One  of  the  best  ways  to 
produce  a credible  report  is  to  follow  the  steps  outlined  above. ^ 


4.  For  a discussion  of  guidelines  for  writing  a report  with 
clarity  and  with  ease  of  understanding  by  the  reader,  see 
Technical  Report  No.  8.  Communications  in  Task  Analysis, 
Training  Manual  IV,  Ch.  3,  pp.  28-73,  "How  to  Write  Clearly”. 
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